Using Real Data to Compare DIF Detection and Effect Size Measures among

To date, many studies have been conducted to compare the performance of different DIF procedures using simulated data sets. However, some results from these simulation studies are inconsistent with one other (e.g., Hidalgo & López-Pina, 2004; Jodoin & Gierl, 2001). This study used real data to systematically investigate the consistencies of DIF detection and effect size among three widely-used DIF procedures: Mantel-Haenszel (MH), Simultaneous Item Bias Test (SIBTEST), and logistic regression (LR). Several indicators, including correlations among DIF procedures effect size measures, matching percentages, and relative matching percentages, were used to evaluate the consistencies among the DIF procedures. The results showed high correlations among DIF effect size measures, moderate to high matching percentages among DIF classifications, and a broad range of relative matching percentages among DIF procedures. Using Real Data to Compare DIF Detection and Effect Size Measures among Mantel-Haenszel, SIBTEST, and Logistic Regression Procedures Differential item functioning (DIF) is of great interest to researchers and educators given that DIF poses a potential threat to test fairness. A variety of DIF detection procedures and effect size measures have been proposed to quantify the magnitude of DIF, such as the IRT methods ( Lord, 1980; Thissen, Steinberg, & Wainer, 1993), the Mantel-Haenszel statistic (MH; Holland & Thayer, 1988), the standardization procedure (Dorans & Kullick, 1986), the Simultaneous Item Bias Test (SIBTEST; Shealy & Stout, 1993a), and Logistic Regression (LR; Swaminathan & Rogers, 1990). Often, multiple procedures are used simultaneously to help detect items showing DIF (Hambleton & Jones, 1994). As a result, high consistencies or matching percentages among DIF procedures are of consequence in real testing situation. To date, many studies have been conducted to compare the performance of different DIF procedures using simulated data sets (e.g., Fidalgo, Ferreres, & Muniz, 2004; Gierl, Jodoin, & Ackerman, 2000; Hidalgo & López-Pina, 2004; Jodoin & Gierl, 2001; Narayanan & Swaminnathan, 1994; Roussos & Stout, 1996). However, some results from these simulation studies are inconsistent with one other. For instance, Jodoin and Gierl (2001) reported that 68.2% of DIF items were identified as containing at least moderate DIF using logistic regression procedure in their simulation study. In contrast, Hidalgo and López-Pina (2004) found that, by using the same procedure and same classification guideline, only 15.3% of DIF items were classified as having moderate DIF. Therefore, the purpose of this study is to use real data to systematically investigate the consistencies of DIF detection and effect size measures among three widely-used DIF procedures: the MH procedure, the SIBTEST procedure, and the LR procedure. This paper is divided into four sections. First, a brief overview of the three DIF procedures is provided. Second, the methods used in this study for evaluating the consistencies among the three DIF procedures are described. Third, the results are presented. Forth, the implications and future directions are discussed. Overview of the three DIF procedures Mantel-Haenszel Mantel-Haenszel (MH) is one of the widely-used approaches for identifying DIF based on analysis of contingency tables (Clauser & Mazor, 1998; Holland & Thayer, 1988). In MH procedure, a chi-square test with one degree of freedom is yielded to test the null hypothesis that there is no relation between group membership and test performance on one item after controlling for ability. MH is computed by matching examinees in each group on their total test scores and then forming a 2-by-2-byK contingency table for each item, where K is the total number of score levels on the matching variable, namely, the total test score. At each score level , a 2-by-2 contingency table is created for each item , as shown in Figure 1. j i The MH chi-square test is calculated as follows: